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Preface 

The National Center on Educational 
Outcomes (NCEO) has been 
developing a conceptual model of 
outcomes and indicators to enhance 
outcomes assessment. It has 
convened groups to define 
terminology, examine the 
fundamental assumptions that 
underlie the process of assessing 
outcomes, develop a model of 
outcomes and indicators, and 
specify outcomes and possible 
indicators. So far, NCEO has 
published: 

Educational Outcomes and 

Indicators for Students 

Completing School 
Educational Outcomes and 

Indicators for Individuals at the 

Post-School Level 
Educational Outcomes and 

Indicators for Early Childhood 

(Age 3) 
Educational Outcomes and 

Indicators for Early Childhood 

(Age 6). 



NCEO works with state and federal 
education agencies to facilitate and 
enhance the development and use of 
indicators of educational outcomes 
that are applicable to all students, 
including students with disabilities. 
While developing its model, NCEO 
has worked closely with educators, 
parents, policymakers, advocates, 
and researchers throughout the 
nation who have given NCEO broad 
consensus on its approach and on 
the model. NCEO is eager to see 
states, districts, and local schools 
capitalize on its efforts by 
collecting outcomes data that can 
help to improve the results of 
education for all students, including 
those with disabilities. 

This Self-Study Guide is written for 
you, whether you are in a state 
department of education, a local 
school district, or a local school. In 
this guide you will find that NCEO 
incorporates what it has learned 
through its national efforts as well as 
some of the experiences of state and 
local personnel who so generously 
shared their expertise. The text 
describes a step-by-step process that 
captures most of what NCEO has 
learned and organizes the 
information into an easy-to- 
implenient handbook. 



Use ihis guide as a companion piece 
to the outcomes and indicators 
booklet series being produced for 
six developmental levels: age 3, age 
6, grade 4, grade 8. school 
completion and post school, all of 
which are available from NCEO. 
However, do not limit yourself to 
the outcomes and indicators in these 
booklets, but use them as a starting 
point. By adding and revising 
outcomes and indicators, you 
address your state/community 
contexts and your specific needs for 
information. 

NCEO's overall goal is to increase 
the availability and use of 
educational outcomes information 
to help you improve programs, 
create better policies, increase 
accountability for outcomes, and 
improve public perceptions of 
education for all children and youth. 
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Introduction 



Purpose and Uses 

This user's guide will help you 
develop a system of outcomes and 
indicators to evaluate educational 
results in your school, district, or 
state. It includes the information 
you will need to develop and 
implement an outcomes-assessment 
program. It takes you step-by-step 
through a process to create a system 
of outcomes and indicators, using 
resource tools developed at NCEO. 
This process is similar to the one 
NCEO used when it developed a 
conceptual model of outcomes and 
indicators and sets of outcomes and 
indicators for six developmental 
levels. 



It is unlikely that your unique state, 
school district, or school would 
simply implement all of the NCEO 
outcomes and indicators at a 
specific level. More than likely, you 
will have to modify them for your 
own system. This guide illustrates 
how to do that using NCEO's 
materials as a resource. 

State Education Agencies (SEAs) 
have begun to use the NCEO model 
of outcomes and indicators in a 
number of interesting ways. For 
example, in one state, SEA 
personnel collect data on the extent 
to which students accomplish the 
instructional objectives specified in 
their individualized education 
program (IEP). SEA expects to 
produce a report on educational 
accomplishments for students with 
disabilities following the outline of 
domains in the NCEO model. 



In another state, educators use the 
NCEO model to develop transition 
IEPs for students with disabilities. 
They require transition teams to 
develop objectives in each of the 
eight outcome domain areas 
specified in the NCEO model. In 
still another state, educational 
personnel group IEP objectives on 
the basis of the NCEO model and 
then use curriculum-based 
measurement approaches to monitor 
progress toward those objectives. In 
yet another state, the SEA holds a 
statewide competition among its 
school districts. The winning 
districts produce report cards on the 
educational progress of students 
with disabilities based on the 
NCEO model. 



Policy Options for Outcomes-based Accountability are explored in a 
document entitled Issues and Options in Outcomes-based Accountability 
for Students with Disabilities, produced by the Center for Policy Options 
in Special Education at the University of Maryland. Among the topics 
addressed are the relationship between general and special education 
outcomes, high-stakes accountability, options for selecting outcomes- 
based accountability systems, and issues in implementing systems of 
data collection. (See Resource Materials on page 44.) 
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Why outcomes and indicators? 

A national shift has occurred in 
business, industry, and human 
services from a focus on process 
(what we do) to results (what we 
cause). This trend is most evident in 
education. The public wants to learn 
about more things than the numbers 
of teachers and their degrees, quality 
of facilities, types of books in the 
libraries and classrooms, length of 
the school day or year, and range of 
educational experiences offered. 
They want to know what effects 
educational experiences have on 
children. They ask, "Arc students 
learning?" and "Does schooling 
have a positive influence on 
children, youth and their 
communities?" 

The resources that schools have and 
the practices that they use are only 
significant to the extent that they 
contribute to the desired results. For 
years educators have collected and 



reported information about 
resources and practices (or "inputs" 
and "processes") without linking 
them to results or outcomes. 
However, knowing that X number 
of dollars has been spent or that the 
curriculum addresses all of the 
national goals for education does 
not help to answer the hard 
question: "Is this effective?" To 
answer that question, educators 
need to know what outcomes are 
expected, to translate those 
outcomes into some measurable 
indicators, and to have a system for 
measuring and reporting the 
indicators. Only then will it be 
possible to know whether those 
outcomes are being achieved. 

Overview of the Steps 

An effective way to develop a 
system of outcomes and indicators 
involves four major steps as 
summarized in the Quick-Reference 



Check List on the next page. The 
pages that follow explain each step 
in more detail. Please note, 
however, that although these steps 
appear in a sequential manner, the 
actual development process is not 
so linear. Instead, it requires 
reconsideration of earlier steps as 
later decisions are made. 

Some Resources 

This Self-Study Guide provides an 
overview of four steps that lead to a 
system of outcomes and indicators. 
If you find that you need more 
assistance than what this guide 
offers, refer to the resource 
materials and the technical 
assistance sources listed on pages 
44 through 49. 
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QUICK-REFERENCE CHECK LIST 

1. Establish a solid foundation for your efforts. 

Involve stakeholders up front 

Decide why you want to measure outcomes 

Define your terms 

Consider your assumptions 

Resolve the fundamental issues in assessment of outcomes 

2. Develop, adopt, or adapt a model* 

Select your approach 

Define your outcome domains 

Define your outcomes 

Define your indicators 

3. Establish a data collection and reporting system. 

Decide where you will get your data 

Develop/adapt data collection and analysis mechanisms 

Decide how you will report and use the information 

4. Install the systeir. 

Create incentives and support for adoption and use 

Prepare staff and the public for the changes 

Evaluate the system as it is implemented 
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Stepl 

Establish a Solid Foundation 
for Your Efforts 
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Involve Stakeholders Up Front 



Stakeholders are quite literally 
everyone who has a "stake" in the 
process of measuring educational 
outcomes: teachers, supervisors, and 
any related services personnel. 
Make certain that you involve all 
stakeholders in the process of 
developing an outcomes model and 
outcomes assessment system for 
your setting. It is especially 
important to involve parents and 
representatives of community 
agencies in setting outcomes to be 
measured. 

Involving all stakeholders on the 
front end of development helps you 
consider the range of perspectives 
and needs that will have to be 
addressed as the system architecture 
begins to emerge. When 
stakeholders participate in the 
development of a system that will be 
used with them they develop a better 



understanding of the rationale for 
and issues in implementing the 
system. They also are more likely to 
support the maintenance and use of 
the system through their informal 
and formal networks. 

NCECTs stakeholder involvement 
efforts used a nomination process. 
NCEO asked advocacy groups, 
governmental agencies, professional 
organizations, and other interested 
parties to nominate peo t le who 
could best articulate their 
perspective as stakeholders. NCEO 
found that a nomination process 
helped to establish a two-way 
communication link. Stakeholders 
brought that group's perspective to 
the planning, became sufficiently 
informed through the process to 
understand the decisions being 
made, and had access to the group's 
meetings, newsletters and other 



vehicles to communicate about the 
decision-making process. In short, 
NCEO tried to be inclusive and 
representative, and at the same time 
picked people with enough 
expertise to be helpful. 

Involving those who will be most 
affected by your efforts will help 
ensure success. However, keep in 
mind that involvement does not 
necessarily mean that your 
stakeholders have to attend 
meetings. There can be several 
levels of involvement, from 
working on a core development 
team to simply being informed. The 
worksheet on page seven might 
help you identify potential 
stakeholders and consider ways to 
involve them in the process of 
developing your outcomes model 
and outcomes assessment system. 



Step 1 Establish a Solid Foundation for Your Efforts 
♦ Involve stakeholders up front 

Decide why you want to measure outcomes 

Define your terms 

Consider your assumptions 

Resolve the fundamental issues in outcomes 

assessment 

Step 2 Develop, Adopt, or Adapt a Model 

Step 3 Establish a Data Collection and Reporting System 

Step 4 Install the System 
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Stakeholcte^lnvolvement 




POSSIBLE STAKEHOLDER 
GROUPS 



Core 
Team? 



Steering 
Committee? 



Other Involvement? 

(None? Just inform?) 



Teachers 
Chapter I 
Early childhood 

English as second language (ESL) 

Middle 

Primary 

Secondary 

Special education 

Vocational education 

Other 
Therapists 

Occupational 

Physical 

Speech 
Principals 

Test and evaluation personnel 
School psychologists 
Curriculum specialists 
Supervisors/Coordinators 
Transition/Work-study specialists 
Parents 

Gifted students 

Chapter I schools 

Students in special education 

Other 

School board members 
Legislators 

Business/industry personnel 
Local/state officials 
People from different cultures 
People from different regions 
Other 



Note: There are some materials listed in the Resource Materials seetion about involving stakeholders in developing evaluation and 
accountability systems. In addition, there are references for some materials that describe consensus-building techniques. 
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Decide Why You Want to Measure Outcomes 



NCEO would like to think that all 
school personnel see the value in 
gathering data on educational 
outcomes. And you probably agree 
or you would not be reading this 
guide. But, why and for what 
purpose? 

You might ask yourself and your 
stakeholders questions like: 

• Why do we want to assess and 
evaluate educational outcomes? 

• How will the data be used? 

• Why is this important? 

There are a number of good reasons 
to assess educational outcomes. 
Among them are: 

• Program Improvement 

• Accountability 

• Public Information 

• Policy Formulation 



Program Improvement 

Data on outcomes or results can 
help to identify strengths and 
weaknesses in instructional 
programs and contribute to 
developing ways to improve those 
programs. Knowing to what degree 
students, schools, and systems 
achieve desired outcomes is an 
essential component in the 
evaluation of instructional practices 
and resources. 

For example, you might be 
concerned that your educational 
system lacks some essential 
outcomes. By using the structure of 
the NCEO outcomes model, you can 
examine a more comprehensive set 
of goals and outcomes for schooling 
so that all students, including 
students with disabilities, will be 
working toward a common set of 
goals. 



Accountability 

Data are collected to document, for 
people in authority, the extent that 
expected outcomes of schooling are 
being achieved. Outcomes 
accountability data are used to judge 
whether a student, school, or local 
or state school system is meeting its 
obligations. States may decide to 
trade compliance with certain rules 
for flexibility in procedures, 
permitting innovation with practices 
at the local level in exchange for 
outcomes accountability. The NCEO 
model and outcomes at various 
levels might be used as the basis for 
an effective sys'em of outcomes 
accountability. 



Step 1 


Establish a Solid Foundation for Your Efforts 




Involve stakeholders up front 




♦ Decide why you want to measure outcomes 




Define your terms 




Consider your assumptions 




Resolve the fundamental issues in outcomes 




assessment 


Step 2 


Develop, Adopt, or Adapt a Model 


Step 3 


Establish a Data Collection and Reporting System 


Step 4 


Install the System 
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Public Information 

Collected data can demonstrate to 
the general public the level to 
which outcomes are being achieved. 
The NCEO model serves as a 
framework for reporting 
information to the public about how 
well the schools serve children and 
about outcomes and results at 
selected developmental levels. 



Policy Formulation 

The NCEO model helps you look at 
possible intended and unintended 
dependent variables in policy 
studies. Outcomes data, usually in 
conjunction with process data, are 
collected to contribute to the 
selection of alternatives for 
establishing a policy. For example, 
you might want to look at the 
effects of a collaborative learning 
program versus a peer tutoring 
program in a variety of outcome 
areas. 



There are many reasons to evaluate 
the outcomes of instruction, and it is 
important that you identify your 
purposes carefully. Well-defined 
purposes help set goals and ensure 
that the collected data are used for 
their intended purposes. A well- 
defined purpose specifies why you 
are collecting outcomes data, what 
you intend to collect in terms of 
level or other focus, who you see as 
the primary user of the information, 
and how you expect the information 
to be used. The worksheet on page 
10 will help you and your 
stakeholders organize your thoughts 
about the purposes of an outcomes 
system. 
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Our Purpose 



Why are we collecting outcomes data? 



What data will we collect? 



Who do we see as the primary audience for data reports? 



How do we expect data to be used? 
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Define Your Terms 



Terms like ''outcomes" and 
"indicators" have multiple meanings 
in professional and popular 
literature. You should come to 
agreement with your stakeholders 
on definitions of these and other 
terms that might cause confusion. 
Doing so will improve 
communication and avoid potential 
misunderstandings later. NCEO 
personnel spent a considerable 
amount of time reaching agreement 
on the terms "outcomes" and 
"indicators." You must decide 
whether you can accept these 
definitions or whether you need to 
redefine the terms. 



Outcomes 

Outcomes are the result of 
interactions between individuals and 
schooling experiences. Results are 
what can/should/does happen when 
a person has educational 
experiences. They may be direct or 
indirect, positive or negative, and 
intended or unintended. 

Indicators 

Indicators are symbolic 
representations of one or more 
outcomes (or inputs, contexts, or 
processes) that can be used in 
making comparisons. Indicators 
provide ways of knowing what 
could/should be looked at to find 
out whether desired results are 
being reached. They can be 
numbers or other representations 
such as test scores, levels of 
participation in activities, or 
perceptions of student 



accomplishments by parents or 
others. They can be used for 
comparisons over time, for 
comparisons to an absolute 
standard, or, as NCEO typically 
uses them, for comparisons among 
and within groups. 

Depending on your context and 
purpose for collecting outcomes 
information, you also will have to 
define other terms. Some that have 
proven difficult in states that NCEO 
has worked with are listed on the 
worksheet. For instance, a 
particularly difficult definitional 
issue arose when NCEO had to 
describe outcomes that enable a 
student to reach the ultimate 
outcomes of schooling, for 
example, accommodation skills. 
Are these "enablers" or 
"prerequisites" or are they also 
"outcomes" as NCEO finally 
decided? You must decide such 
issues for yourself. 



Step 1 Establish a Solid Foundation for Your Efforts 

Involve stakeholders up front 

Decide why you want to measure outcomes 
♦ Define your terms 

Consider your assumptions 

Resolve the fundamental issues in outcomes 

assessment 

Step 2 Develop, Adopt, or Adapt a Model 

Step 3 Establish a Data Collection and Reporting System 

Step 4 Install the System 
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Stakehold^lnvolvement 



ESSENTIAL TERMS 



Outcome 



How we will use them 



Suggestion: The result of interactions between individuals and schooling experiences. 
Our definition: 



Indicator 



Suggestion: A symbolic representation of one or more outcomes (or inputs, contexts, 
or processes) that can be used in making comparisons. 
Our definition: 



Other terms that might need discussion and/or definition: 

Accountability, all students, anchors, assessment, benchmarks, content standards, criterion referenced tests, curriculum, 
curriculum alignment, curriculum framework, demonstrators, domain, enablers, evaluation, fundamental skills, 
learning goals, measurement, outcomes-based/-focused/-driven education, performance measurement, performance 
tasks, performance standards, prerequisites, proficiency, rubrics, school performance, scope and sequence, system 
outcomes, testing, valued outcomes, world class. 



OTHER TERMS 
CRITICAL TO US 



How we will use them 



FRir 
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Consider Your Assumptions 



Any system of outcomes and 
indicators is based on a number of 
assumptions. 

After a considerable amount of 
time, NCEO reached agreement on 
the fundamental assumptions that 
underlie the outcomes and 
indicators model and its use. The 
assumptions listed on the worksheet 
and described in detail in NCEO's 
Working Paper 2: An Evolving 
Model of Educational Outcomes for 



Children and Youth with 
Disabilities, were reviewed by other 
professionals, parents, advocacy 
groups, representatives of 
professional associations, and 
policymakers. 

You should examine the extent to 
which your stakeholders agree with 
these assumptions, whether they 
share additional assumptions, and 
whether these need to be modified 
for your purpose. 



Step 1 Establish a Solid Foundation for Your Efforts 

Involve stakeholders up front 
Decide why you want to measure outcomes 
Define your terms 
♦ Consider your assumptions 

Resolve the fundamental issues in assessment 
of outcomes 

Step 2 Develop, Adopt, or Adapt a Model 

Step 3 Establish a Data Collection and Reporting System 

Step 4 Install the System 
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Assumptions 

Which of these nine NCEO assumptions can serve as the foundation for our development efforts without change? 

] . A model of outcomes is needed for all students, and at the broadest level, should apply to all students 

regardless of the characteristics of individuals. 

2. A model of outcomes should primarily focus on intended outcomes, but be sensitive to unintended outcomes 

as well. 

3. A model of outcomes should include both direct and indirect outcomes. 

_ 4. Indicators of outcomes for students receiving special education services should be related, conceptually and 
statistically, to those identified for students without disabilities. 

_ 5. Indicators should reflect the diversity of gender, culture, race and other characteristics of the students in 
today's school population. 

6. While indicators should meet research standards, those that do not could still be used. 

7. A comprehensive system of indicators should provide data needed to make policy decisions at the national, 

state, and local levels. 

8. A comprehensive system of indicators should be based on demonstrated functional relationships between 

outcome indicators and indicators of educational inputs, and contextual characteristics and processes; however, 
valued indicators may be included even if functional relationships have not been established. 

9. A comprehensive system of indicators should be flexible, dynamic, and responsive to review and criticism. It 

should also change to meet identified needs and future developments in the measurement of inputs, contexts, 
processes, and outcomes. 

Which assumptions can we edit to be acceptable to our stakeholders? (Make the changes in the text above.) 



What additional assumptions do we share? (List them below.) 



18 

14 



SELF-STUDY GUIDE TO EDUCATIONAL OUTCOMES AND INDICATORS 



NCEO 



Step 2 

Develop, Adopt, or 
Adapt a Model 
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Select Your Approach 



The outcomes assessment process 
needs to be driven by a conceptual 
model that shows how your 
educational system should work. 
The current NCEO model is shown 
on the next page. An extensive 
consensus-building process has 
helped design the model and, if your 
results from Step 1 match the 
decisions that NCEO made, you 
should be able to apply this model 
in your setting with only minor 
changes. If your group's 
assumptions differ from NCEO's or 
if your group resolved the 
fundamental issues in a different 
way. you might want to review the 
Resource Materials or Sources for 
Technical Assistance on pages 44 
and 47 for alternatives. 



The NCEO model suggests that 
resources (input and context) 
influence educational opportunity 
and process, which in turn influence 
educational outcomes. NCEO has 
divided educational outcomes into 
outcome domains. As explained in 
NCEO documents, the model is then 
extended by identifying specific 
outcomes for domains, indicators of 
those outcomes, and possible 
sources of data for those indicators. 

Your model might be more or less 
complex than this one. Depending 
on your purpose, you might find it 
important to provide more detail 
about the contexts, resources, 
opportunities, and processes that 
lead to the desired or expected 
outcomes. 



The worksheet on page 20 will help 
you through the process of deciding 
whether you want to develop a new 
model, adopt the NCEO model, or 
adapt the NCEO model to meet your 
needs. 



Step 1 


Establish a Solid Foundation for Your Efforts 


Step 2 


Develop, Adopt, or Adapt a Model 




• Select your approach 




Define your outcome domains 




Define your outcomes 




Define your indicators 


Step 3 


Establish a Data Collection and Reporting system 


Step 4 


Install the System 
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NCEO 



This conceptual model shows the 
complete educational process, with 
Resources (Input and Context) 
influencing Educational 
Opportunity and Process. These, in 
turn, influence the Outcome 
Domains (the shaded areas), which 
have a return influence on both 
Resources, and Educational 
Opportunity and Process. 



Two of the shaded domains, 
Presence and Participation, and 
Accommodation and Adaptation, 
are placed next to Educational 
Opportunity and Process. This 
reflects their greater association 
with process variables even though 
they are measured as outcomes. 



Conceptual Model of Outcomes 
School Completion 



♦ 



= OUTCOME DOMAIN 



♦ 



Presence and 
Participation 



Resources 
(Input and Context) 



Educational 
Opportunity and 
Process 





Accommodation 
and Adaptation 




Physical Health 



Responsibility and 
Independence 



Contribution and 
Citizenship 



Academic and 
Functional Literacy 



-1 



Personal and Social 
Adjustment 



Satisfaction 
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NCEO 



Our Conceptual Model 




Are our purpose, assumptions, and decisions about fundamental issues sufficiently consistent with those of NCEO that 
we can adopt the NCEO conceptual model as shown on page 19? (If yes, stop here and go to next step.) 



If we cannot adopt the NCEO model, can we adapt it? If so, to what extent must we modify the NCEO picture of 
resources, opportunities, processes, and outcomes? 



If we change the model, what should we add or take away to better describe how our education system works? 



Sketch of our revised/adapted model: 
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NCEO 



Define Your Outcome Domains 



As a major step in creating a 
conceptual model you will have to 
specify the domains that define the 
outcomes that you expect your 
educational system to produce or 
that you want to track. NCEO has 
considered many outcome domains, 
including the possibility of using the 
National Education Goals as a 
framework. 

No single framework met all of 
NCEO's assumptions, especially the 
need to be sufficiently inclusive to 
encompass the full range of social 
and life role outcomes in addition to 
academic outcomes. NCEO had to 
develop a framework that 
incorporated all of the other 



approaches and still spanned the 
developmental levels. With the help 
of stakeholders, NCEO chose the 
following six outcome domains: 

• Physical Health 

• Responsibility and 
Independence 

• Contribution and Citizenship 

• Academic and Functional 
Literacy 

• Personal and Social Adjustment 

• Satisfaction 

NCEO also identified two other 
domains that remain objects of 
debate as ''outcomes" - Presence 
and Participation, and Accom- 
modation and Adaptation. It linked 
these two areas to the Educational 



Opportunity and Process portion of 
the conceptual model because some 
stakeholders viewed them as interim 
or enabling outcomes rather than as 
targets of education. However, most 
stakeholders viewed them as critical 
outcomes, especially for students 
who historically have been 
disenfranchised, such as students 
with disabilities, ESL students, and 
Chapter I students. 

You will have to determine to what 
extent you need to include those 
types of outcomes in your model. 
The worksheet on page 22 is 
designed to help you define your 
outcome domains. 



Step 1 


Establish a Solid Foundation for Your Efforts 


Step 2 


Develop, Adopt, or Adapt a Model 




Select your approach 




# Define your outcome domains 




Define your outcomes 




Define your indicators 


Step 3 


Establish a Data Collection and Reporting System 


Step 4 


Install the System 
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Conceptual Model of Educational Outcomes 



Do the six primary NCEO outcome domains generally reflect the expectations of our educational system? Are these 
the areas that we need to track? What needs to be added or removed to reflect our purposes? 

Physical Health - Healthy behaviors, attitudes, and knowledge related to physical health. 

Responsibility and Independence - Behavior that reflects the ability to function independently and assume 

responsibility for self. 
Contribution and Citizenship - Return to society and participate as citizen in society. 

Academic and Functional Literacy - Use of information tc function in society, to achieve goals, and to develop 
knowledge. 

Personal and Social Adjustment - Socially acceptable and healthy behaviors, attitudes, and knowledge 

regarding mental well-being. 
Satisfaction - Favorable attitude toward education. 



Should we track these outcomes in the educational opportunity and process arena? 

Presence/Participation - An individual's presence in a particular setting and the extent to which meaningful 
participation occurs. 

Accommodation and Adaptation - Modifications made to allow other accomplishments. 



What are the domains for our system? How will we define them? 
Domains Definitions 
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Define Your Outcomes 



You will have to define the 
outcomes in each domain before 
they can be easily communicated. 

NCEO first defined outcomes in a 
way that possibly could apply 
across all developmental levels by 
drafting a broad definition of each 
domain. But as work began with 
groups, it became clear that more 
traditional statements needed to be 
produced. 

While attempting to clearly 
communicate the expectations of the 
domain, NCEO avoided producing 
statements that were limiting and 



would not apply to all ages and • 
groups. As NCEO's efforts 
continued, some outcomes were 
excluded from certain 
developmental levels, while others 
were added. 

You will have to decide whether to 
adopt or adapt NCEO's outcome 
statements or develop another set. 
Again, your choices will depend on 
your earlier decisions. You also will 
have to decide whether you want to 
use a single set of outcomes for all 
developmental levels and apply 
other rules to reflect your decisions 
about the fundamental issues raised 



in step one of the process. These 
issues are: 

• Single system versus separate, 
related systems 

• Whether to differentiate based 
on age/grade levels 

• Categorical versus non- 
categorical indicators 

• Whether to differentiate based 
on ability 

• System versus individual 
indicators 

• Whether to consider enablers 
and prerequisites as outcomes. 

On page 24, you will find a 
worksheet to help you document 
answers to these questions. 



Step 1 


Establish a Solid Foundation for Your Efforts 


Step 2 


Develop, Adopt, or Adapt a Model 




Select your approach 




Define your outcome domains 




• Define your outcomes 




Define your indicators 


Step 3 


Establish a Data Collection and Reporting System 


Step 4 


Install the System 
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Outcomes 



Will we include the following NCEO outcomes? Will we follow the NCEO approach and develop outcomes specific to 
each developmental level? 



Include? 



Levels? 



Al 
A2 
A3 



Bl 
B2 



CI 

C2 



Dl 

D2 



El 
E2 
E3 



NCEO Domains (A-H) and Outcomes at the School Completion Level 

A. Presence and Participation 

Is present in school 
Participates 
Completes school 

B. Accommodation and Adaptation 

Makes adaptations, accommodations, or compensations necessary to achieve 

outcomes in each of the major domains 
Demonstrates family support and coping skills 

C. Physical Health 

Makes healthy lifestyle choices 

Is aware of basic safety, fitness, and health care needs 

D. Responsibility and Independence 

Gets about in the environment 
Is responsible for self 

E. Contribution and Citizenship 

Complies with school and community rules 

Knows the significance of voting and the procedures to register and vote 
Volunteers 



Fl 

F2 

F3 
F4 



Gl 

G2 
G3 
G4 



HI 

H2 
H3 



F. Academic and Functional Literacy 

Demonstrates competence in communication 

Demonstrates competence in problem-solving strategies and critical thinking 
skills 

Demonstrates competence in math, reading and writing skills 
Demonstrates competence in other academic and nonacademic skills 

G. Personal and Social Adjustment 

Copes effectively with personal challenges, frustrations, and stressors 

Has a good self image 

Respects cultural and individual differences 

Gets along with other people 

H. Satisfaction 

Student satisfaction with high school experience 

Parent/guardian satisfaction with the education that students receive 

Community satisfaction with the education that students receive 
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NCEO 



Define Your Indicators 



show all of the indicators at all of 
the levels. All six documents may 
be ordered to accompany this guide. 
Here are two examples from the 
school completion level: 
Domain E: Contribution and 
Citizenship. 

Outcome E3: Volunteers. 
Indicator a: Percent of students 
who volunteer time to school, civic, 
community, or nonprofit activities. 

Domain F: Academic and 
Functional Literacy 
Outcome F3: Demonstrates 
competence in math, reading, and 
writing skills. 

Indicator e: Percent of students 
who demonstrate competence in 
writing necessary to function in 
their current home, school, work, 
and community environments. 

For each outcome, you might have 

several indicators. For example, be 

NCECTs stakeholders had identified co 

five other indicators for outcome F3 pa 

- demonstrates competence in math, in< 



Establish a Solid Foundation for Your Efforts 



Your outcomes must be translated 
into indicators, which are symbolic 
representations that can be used in 
making various kinds of 
comparisons. Over time, these 
comparisons can be made to an 
absolute standard, or, between and 
within groups. Indicators provide 
information on whether desired 
results are being reached. Using 
NCEO's definition, indicators can 
be numbers or other representations. 
You will want to review the 
assumptions that you adopted with 
your stakeholders. NCECTs 
assumptions will lead you in very 
specific directions regarding such 
things as measurability (see step 1 ). 

NCEO has elected to develop 
indicators specific to each of six 
developmental levels. Use of this 
approach allows you to track the 
same outcome domains for students 
at different ages using 
developmentally relevant 
information. Unfortunately, there is 
not enough space in this guide to 



Step 1 
Step 2 



Step 3 
Step 4 



Develop, Adopt, or Adapt a Model 

- Select your approach 

Define your outcome domains 
Define your outcomes 
• Define your indicators 



reading, and writing skills - for 
students completing school. 
You will not be able to collect data 
on every possible indicator at every 
level. You will have to go through a 
process of brainstorming possible 
indicators and then narrowing down 
the list to those that are most 
appropriate for your system and 
most meaningful for your purposes. 

To develop indicators, NCEO asked 
experts in a variety of areas to 
suggest potential indicators at each 
of the levels. These were compiled 
and edited into similar language for 
use with teams of stakeholders at 
each level who worked through a 
consensus-building process to select 
the indicators that best reflected the 
outcome domain and the outcomes. 
As appropriate, indicators were 
edited further so they clearly 
scribed how the outcomes would 
represented during the data 
Election phase. The worksheet on 
ige 26 will help you examine the 
dicators that you define. 



Establish a Data Collection and Reporting System 
Install the System 
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S*p2 

Indicators 

Check your indicators as you proceed through this step, perhaps using this worksheet with your stakeholder group(s): 
Are our indicators consistent with our assumptions? 

If we collected information using these indicators, would we fulfill the purpose we stated? 



Have we considered a full array of indicators for each outcome domain and each outcome in our model? Are there any 
outcomes without indicators? 



Is each indicator inclusive? Can we obtain that information on all students? If not, can the indicator be changed to be 
more inclusive? 



Have we narrowed down the indicators through a consensus-building process that will allow us to target the most 
important information that our stakeholders want about the outcomes? 



Have we focused on what is really important, listing significant and needed indicators before determining whether they 
can be collected? 



Will our final set of indicators meet the criteria that experts apply to evaluation and data systems? 

USEFUL Provides practical information for our intended audiences. 

FEASIBLE.... Results in a system that is realistic, prudent, diplomatic and frugal. 

PROPER Is legal and ethical regarding those involved and affected. 

ACCURATE... Reveals and conveys technically adequate information for our purpose. 
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NCEO 



Step 3 

Establish a Data Collection 
and Reporting System 
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Decide Where You Will Get Your Data 



Before you document your status on 
indicators, you must have each 
indicator linked to at least one data 
source. One of NCEO's hopes is that 
available data will be used rather 
than developing new data systems. 
Therefore, you need to look at 
existing systems to determine 
whether they contain information on 
the indicators that stakeholders have 
selected. 

You might start by asking yourself, 
"What information are we already 
collecting that is consistent with this 
indicator?" Among the types of 
school, district, and state sources 
that could be considered are survey, 
interview, and evaluation results, 
data bases, individual files, and 
other less systematic data such as 
meeting summaries. Information 
available in the community, such as 
employment rates, also can help in 
judging outcomes. 



Next you can prepare a list of 
matches and misses. Not every data 
source will be a perfect match. At 
the beginning stages of system 
development you will want to 
include existing sources thai 
approximate the indicator while you 
are developing data systems that are 
more specific. 

For example, to measure 
achievement of a writing outcome 
you might want a portfolio that 
includes a direct measure of student 
writing skills in home, school, work, 
and community environments. 
However, you might have to settle 
for the results from your district- 
wide on-demand writing tests or 
more limited writing portfolios. 
Later, you could find or develop 
more ways to document the full 
indicator. In the meantime, you will 
be able to demonstrate that Cat 
indicator was not ignored. 



In many cases, you may find that 
data already exist, but not in a form 
that is easily accessible. An example 
would be a hand written note in a 
student file or a file maintained by 
the police department or social 
services. List all such data sources 
anyway during the development 
phase, and then investigate whether 
such data can be made appropriately 
accessible for your use. 

Some of the data sources you might 
consider are listed on the worksheet 
on page 29. And, you might find 
that you have identified data sources 
that you need but that do not yet 
exist. These items are candidates for 
future data collection systems. 



Step 1 Establish a Solid Foundation for Your Efforts 

Step 2 Develop, Adopt, or Adapt a Model 

Step 3 Establish a Data Collection and Reporting System 
▲ Decide where you will get your data 

' Develop/adapt data collection and analysis mechanisms 

; . Decide how you will report and use the information 

Step 4 Install the System 
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NCEO 




Have we considered the following school and school 
on our indicators? 

Data bases/compiled records: 

a Attendance, dropout, and graduation records 
a Budgets 

a Club/organization membership lists 

a Discipline records 

a End-of-course testing records 

a Grades 

a Portfolio ratings 

a School or system-wide test results 

and national testing 
a Student tracking system data 

Individual records: 

a Individual student files, e.g., IEP files 
a Personnel records 
a Therapy notes/records 



i data as we have checked for matches and misses 

Survey/interview data from: 
a Parent/public 

a Policymakers/board members/legislators 
a Students 
a Staff 

a Work-study employers 

Records that might not be available in easily 
accessible form: 

a Correspondence files 

a Evaluation reports from outside groups 

a Minutes of board meetings 

a Minutes of PTA meetings 

a Observation notes 

a Site-based Management Council minutes 
a Videotapes: classes, playgrounds, club 

activities, student projects 
a Work samples 



Have we considered the following examples of non-school records as we have checked for matches and misses on 
our indicators? 
a Census data 

a City records, i.e., regarding housing, employment, abuse, health, etc. 

a Hospital, child care, recreation, volunteer, or other community records 

a Newspaper articles/data on education or social issues 

a State planning department records 

a Vandalism/juvenile detention/corrections records 

a Voter and driver registrations 
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NCEO 



Develop/Adapt Data Collection 
and Analysis Mechanisms 



It is likely you will have to create 
new data collection mechanisms to 
address new indicators or to include 
populations that you have not 
included before. Your data 
collection systems need to be 
sensitive to cultural differences 
during sampling, instrument 
development, data collection, and 
analysis. 

Check existing data for accuracy 
and completeness. Ask yourself 
whether students with unique 
characteristics (for example, 
students with disabilities, ESL 
students. Chapter I students) were 
included. 

When NCEO studied state and 
national testing systems, it found 
that as many as 50 percent of 
students with disabilities were 
excluded from testing. Often the 
exclusion was based on a category 
of disability or placement rather 
than being based on individual skills 
and needs. 



Many assessment systems are now 
being made more inclusive. This 
involves being more systematic 
about including students in testing, 
providing consistent adaptations and 
accommodations, and providing 
alternative measures for students 
whose curriculum differs from the 
general curriculum (students with 
significant cognitive disabilities). 
For instance, Kentucky has an 
alternative portfolio system for 
students with severe disabilities so 
that these portfolios can be included, 
with the regular portfolios in the 
total picture of a school and district. 

To decide what form your data 
should take, see NCEO's indicator 
statements for help. You will find 
that most of NCEO's indicators deal 
with percentages and rates. Com- 
parisons over time will be an 
important future component of your 
system requiring yearly 
maintenance of data in a consistent 
format. Comparisons between 
groups require obtaining 
information from another source on 
a comparable population (like from 



a district or state or from national 
statistics). Comparisons to an 
absolute standard require a formal 
standard-setting process. 

Some states are separating data 
based on non-standard testing or 
non-standard collections from the 
total pool, if you choose this 
method, you will want to find other 
ways to demonstrate how all 
students did on each outcome. 

Developing new data collection 
instruments and procedures requires 
a carefully considered process of 
design, development, field testing, 
revision, sampling, training, 
collection, error checking, data 
entry, and analysis. Quantitative 
(numbers) and qualitative (words) 
procedures demand significant 
expertise. Consult with an expert 
when you decide to develop new 
information components. Sources in 
the section Sources for Technical 
Assistance can give you guidance. 
Some of the issues in data collection 
that you will want to discuss are 
included on page 31. 



Step 1 Establish a Solid Foundation for Your Efforts 
Step 2 Develop, Adopt, or Adapt a Model 

Step 3 Establish a Data Collection and Reporting System 

Decide where you will get your data 

A Develop/adapt data collection and analysis mechanisms 

Decide how you will report and use the information 

Step 4 Install the System 
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Stop 3 



Data Collection and Analysis 



Are the existing data sources we selected inclusive of all in terms of: 
A Instrument design and field testing? 
A Sampling? 
A Data collection? 

A Allowed accommodations and adaptations? 
A Analysis? 

Are the existing data free from significant errors? 



Are the existing data accessible to us for re-analysis and long-term storage if necessary? 



Do we have the expertise to develop the new data collection systems that we need? 



Have we accounted for all students in our instrument design, development, field testing, revision, sampling, training, 
collection, error checking, data entry, and analysis plans? 



Do our data reflect observed behavior and valid, reliable measures to the greatest extent possible? 



Is our projected total data burden worthwhile for the information we are obtaining? Are the data feasible to collect and 
maintain? 



Is our data collection and analysis still focused on our purpose, or have we drifted? Will it help our audicnce(s) make 
decisions? 
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Decide How You Will Report and Use the Information 



How you report on outcomes should 
be driven entirely by your stated 
purpose. The key criteria involve 
addressing the information needs of 
your intended audience(s) and not 
violating the agreements understood 
in your plan for data collection. 

Most likely your audience is 
someone other than yourself or your 
staff. That means that you will have 
to ask the intended audience what 
would help it make decisions 
consistent with your stated purpose 
(like program improvement, 
accountability, public information, 
or policy formulation). You can 
show the audience some alternative 
mock tables and charts and simulate 
a reporting/decision-making session. 
It is helpful to do this even before 
you collect data to avoid collecting 
too much or too little information. 



Probably the most important 
decision to be made deals with the 
use of the outcomes data. Uses that 
have what are called "high stakes" 
can have negative consequences on 
subsequent data collection. High 
stakes exist when individuals or 
school systems receive rewards or 
sanctions based on the results. The 
mere release of information 
comparing one district or one school 
with another can be considered high 
stakes. 

One state studied this phenomenon 
and found that high stakes reporting 
led to increased referrals of students 
to special education programs and 
increased retention of students in 
grade levels that were not being 
tested. 



Ethics requires that reporting and 
using the data should be consistent 
with your original purpose. 
Violation of the pact made at the 
time of data collection can have 
devastating long-term effects for 
your system. Careful editing, 
inclusion of cautions, and controlled 
release of reports is essential to 
ensure attention to your purpose. 

Other reporting decisions you will 
have to make deal with levels of 
reporting, types of comparisons to 
be reported, formats and types of 
reports, presentation of data, ways 
of grouping data, range of 
audiences, vehicles for 
dissemination, and year-to-year 
linkages. Some of these decisions 
are explained in greater detail in the 
worksheet on page 33, which will 
help you decide how you will report 
and use outcomes data. 



Step 1 Establish a Solid Foundation for Your Efforts 
Step 2 Develop, Adopt, or Adapt a Model 

Step 3 Establish a Data Collection and Reporting System 

' Decide where you will get your data 

Develop/adapt data collection and analysis mechanisms 
A Decide how you will report and use the information 

Step 4 Install the System 
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Resolve the Fundamental Issues in 
Outcomes Assessment 



As the model developed, NCEO 
was forced to address the following 
fundamental issues that you, too, 
will have to address: 

• Single system versus separate, 
related systems 

• Whether to differentiate based 
on age/grade levels 

• Categorical versus 
non-categorical indicators 

• Whether to differentiate based 
on ability 

• System versus individual 
indicators 

• Whether to consider enablers 
and prerequisites as outcomes 

NCEO chose to develop a single set 
of outcomes that could be used for 
all students. To begin with, age 
levels were selected consistent with 
the major data collection systems in 
the nation, primarily the National 
Assessment of Educational Progress 
(NAEP) and the work of the 
National Education Goals Panel 



(NEGP). Therefore, NCEO chose 
age 6, grade 4, grade 8, and grade 
12 (school completion) as key 
levels. However, because of the 
national emphasis on early 
childhood and the special need for 
information on the outcomes of 
early intervention services, age 
three was included. In addition, 
NCEO believed it was essential to 
look at post-school status in order to 
address concerns of groups like the 
Secretary's Commission on 
Achieving Necessary Skills 
(SCANS) and to determine long- 
term effects. 

NCEO asked stakeholders about 
categorical versus non-categorical 
indicators and, while there was still 
some disagreement, the majority 
thought that statements of inclusive 
outcomes communicate positively 
about inclusive systems of 
education. Nevertheless, some states 
looked at establishing different 



levels of expectation, even if the 
students were working toward the 
same outcomes. 

On the issue of system versus 
individual student outcome 
indicators, NCEO has elected to 
include both. Some stakeholders 
still argue, however, that the focus 
should be on individual student 
accomplishments and not on system 
accomplishments (dropout rates or 
percentage of students graduating). 

NCEO has discussed these issues 
with various groups and individuals, 
and has found that there is never 
total agreement on the answers. 
You will need to engage in a 
discussion of the issues, attempt to 
reach consensus regarding them, 
and talk about how they affect the 
purpose and design of your 
assessment program. The questions 
on the following worksheet (page 
16) might be useful in your 
discussions. 



Step 1 Establish a Solid Foundation for Your Efforts 

; . Involve stakeholders up front 

Decide why you want to measure outcomes 
Define your terms 
Consider your assumptions 
♦ Resolve the fundamental issues in outcomes 
assessment 

Step 2 Develop, Adopt, or Adapt a Model 

Step 3 Establish a Data Collection and Reporting System 

Step 4 Install the System 
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Fundament?! Issues 

Will we develop a single system of outcomes and indicators to apply to all programs and all students or will we 
develop separate systems for separate programs? 



If we develop separate systems: 

♦ How will they be interrelated? 

♦ Will our system be further broken down by category of student, e.g., would students with different types of 
disabilities have different expected outcomes and different indicators? 

♦ Would students with different levels of severity of disability have different expected outcomes? 



At what age or grade levels will we define outcomes and collect outcomes data? 



Will we have system-level outcome indicators as well as indicators of individual student learning? 



Will we have only ultimate educational outcomes or also include necessary prerequisites that we want to track such as 
presence, participation, accommodations, and modifications? 
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Reports and Uses 



Do our reporting formats meet the needs and expectations of our audiences? 



Have we considered the implications of high stakes uses of our outcomes data? 



Is our plan for reporting and use consistent with our stated purpose? 



Have we considered the following: 

A Level(s) of reporting — student, classroom, school, district, region and/or state. 

A Types of comparisons to be reported — over time, against comparable groups, against benchmarks. 
A Format of report(s) — printed, verbal, slide/tape, videotape/TV. 
A Presentation of data — tables, graphs, charts, pictures, stories. 

A Ways of grouping data — by gender, race, disability, age, region, opportunity to learn. 

A Types of reports — brochures, executive summaries, abbreviated sets of tables, full technical reports. 

A Range of dissemination — all school personnel, parents, the public, the media, other like 
entities (other schools, states, or districts). 

A Vehicles for dissemination — through an intermediary such as a principal, via a major media event, in 
conjunction with an action planning activity. 

■ Year-to-year linkages — consistent logo, color, or cover format that can be used every year, maintenance of 
tables and charts for access, and updates in subsequent years. 
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Create Incentives and Support for Adoption and Use 



A system of outcomes and 
indicators cannot be installed 
overnight. Those who implement 
and use the information will need to 
see personal and programmatic 
benefits before the system can be 
considered fully in place. Therefore, 
you must have incentives for the 
teachers, parents, and administrators 
who will ultimately ensure the 
success of the system. Also, you 
must provide long-term support to 
maintain the system. 

Two commonly used incentives are 
public comparisons and sanctions 
for failure to meet standards or 
goals. Public comparisons formally 
display schools, districts, or states 
side-by-side. Sanctioning involves 
negative techniques such as 
withdrawal of accreditation, 
takeovers of schoois, and reduction 
of funding based on identification of 
less-than-adequate outcomes. Both 



comparisons and sanctions represent 
"high stakes" uses of the system. 
They can lead to overemphasis on 
appearances, even without 
substantive changes. 

You can use at least five approaches 
as positive incentives for adoption 
and use. The following techniques 
may be used independently or 
together: 

Fund research and demonstration 
projects as pilots* 

You can fund projects at schools 
throughout the state or district to 
work out the bugs in an outcomes 
measurement system and let the 
schools serve as sites where other 
district or school personnel can visit 
and learn from their peers. This 
approach allows adopters to make 
the choice of adoption rather than 
being forced into using the system. 



Establish recognition and 
networking systems. 
You can establish recognition and 
networking systems for schools and 
districts that begin io install the 
outcomes measurement system and 
use it to help improve their 
programs. For example, you can 
create an appeal for others to join 
the leaders in the district or state by 
using newsletters and electronic 
message systems. These can 
describe how the system is being 
installed and used in various places 
(but not list their specific results). 

Or, you can support meetings of 
system managers and users for the 
purpose of discussing issues and 
learning about extended options. 
This approach can become an 
especially powerful incentive if you 
involve parents in looking at the 
activities in other schools or 
districts. 



Step 1 Establish a Solid Foundation for Your Efforts 

Step 2 Develop, Adopt, or Adapt a Model 

Step 3 Establish a Data Collection and Reporting System 

Step 4 Install the System 

■ Create incentives and support for adoption and use 

Prepare staff and the public for the changes 

Evaluate the system as it is implemented 
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Market the system. 

By using marketing techniques 
targeied at specific audiences, you 
can stimulate adoption and use of an 
outc }mes measurement system. 
First, study the needs of each 
audience and then, communicate 
how the system responds or can 
respond to those needs. 

Provide resources for some of the 
installation or use. 

Providing some of the human, 
equipment, materials, and fiscal 
resources needed for installation and 
operation of the system can increase 
the number of adopters rapidly, even 
if the amount of support is limited. 
For example, you can provide mini- 
grants to use for installation 
purposes (like, staff training or 
purchasing consultant services), 



provide some of the needed 
hardware or software, provide 
instrumentation or guidelines, or 
provide data analysis support for 
site-specific uses. 

Provide technical assistance. 

You can offer sites the use of 
consultants in content areas or with 
computer or evaluation skills for a 
specified number of days after 
initial training and installation. At 
the discretion of the districts and 
schools, these consultants can guide 
sites through difficult areas and help 
them deal with context-specific 
problems. This approach leaves site 
leaders feeling supported and 
increases the chances of initial 
successes. (See the list Sources for 
Technical Assistance on page 47 for 
information on consultants.) 



Additionally, it is essential that you 
arrange long-term support for 
ongoing maintenance, technical 
support, and use at the center where 
information will be handled. You 
will need commitment of staff time, 
hardware, software, and materials as 
well as commitment from decision 
makers to use the outcomes 
indicator information after it 
becomes available. The worksheet 
on page 38 provides the guiding 
questions for you to consider on 
incentives. 
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What incentives are we prepared to offer singly or in combination to increase our chances of wide-scale adoption and 
use without negative side effects? 



What provisions have we made to maintain the central system after it is installed? 
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Prepare Staff and the Public for the Changes 



You may find it difficult to change 
to an approach that focuses on 
outcomes rather than inputs and 
processes. It entails installing a 
system designed to document how 
well you do on a system-wide basis 
and requires both technical and 
affective changes. Therefore, a staff 
development plan becomes 
essential. 

You can expect staff and parents to 
go through a variety of predictable 
stages: lack of interest or under- 
standing, initial awareness and 
knowledge, decision to act, initial 
implementation, comfortable use, 
and adaptation and innovation. 
Since your staff will go through 
these stages at different speeds, be 
ready to provide them with training 
when needed. Also,- remember that 
it is as important to train in-house 
staff as it is to train those in the 
cooperating sites - from secretaries 
to the school board chairperson. 
Additionally, there should be a plan 
to orient the general public and the 
media. 



In the initial stages, you will need to 
convince staff and parents that they 
will not experience significant 
personal loss as the system is 
installed and that personal benefits 
will outweigh any losses. It is too 
early at this point to discuss such 
things as ultimate benefits to the 
community. Instead, concentrate on 
communicating to them about such 
things as how much time and effort 
it will take and how the results will 
be used. They will be concerned 
about the system's advantages over 
what they are currently using, how 
compatible it is with current 
systems and curricula, how complex 
it will be to install and maintain, 
whether they have to do it all at 
once or can install it in stages, and 
how immediately apparent the 
results will be to others. 

After your people have decided to 
be participants, you will want to 
arrange for extensive training in the 
mechanics of gathering, reporting, 
and/or using the information 
(depending on their roles). At this 



point your focus should be on the 
"how to" of the system more than 
on the rationale and the 
implications. Plan more than a one- 
time training session for your 
various groups. Providing follow- 
along training and technical 
assistance will significantly increase 
your chances of success. 

Early adopters who become 
accustomed to the system will be 
able to train others. As a bonus, you 
will begin to learn from them as 
they discover how to make the 
system meet additional needs, how 
to be innovative with the way the 
system operates in their sites, and 
how to use the information. If you 
create or suggest a system for 
ongoing idea exchanges, you can 
help move the adoption process 
along and can encourage people to 
stay with the system. Several 
questions to help you focus staff 
development efforts are included in 
the worksheet on page 40. 



Step 1 


Establish a Solid Foundation for Your Efforts 


Step 2 


Develop, Adopt, or Adapt a Model 


Step 3 


Establish a Data Collection and Reporting System 


Step 4 


Install the System 




Create incentives and support for adopti and use 




■ Prepare staff and the public for the changes 




Evaluate the system as it is implemented 
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Staff I 




ment 



Based on the stakeholder list in step #1 , what concerns will each stakeholder have about the installation of this 
system? How can we help each individual deal with her/his concerns about the following: 

■ Relative advantages 

■ Compatibility 

■ Complexity 

■ Potential for incremental installation 



What training can we provide to meet the needs of individuals as they progress through the following stages of 
adoption at different speeds: 

■ Initial awareness and knowledge 

■ Decision to act 

■ Initial implementation 

■ Facile or comfortable use 

■ Adaptation and innovation 

How will we ensure ongoing follow-along and technical assistance as ii. plementation proceeds? 



Visibility 



\ 



Will we provide for cross-unit networking? 



Have we considered the needs of internal staff and policymakers as well as external staff? 



Do we have a plan for orienting and training parents, the community, and the media? 
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Evaluate the System as it is Implemented 



As you implement your system, 
gather information to help you 
continually evaluate and improve it. 
Your answers on the worksheets in 
this Self-Study Guide can serve as 
your standards. 

Some evaluators find it helpful to 
ask questions at different stages, 
collect data to answer the questions, 
and then use the answers to make 
changes before going on to the next 



step. Sample questions for overall 
evaluation of your system are listed 
in the worksheet on page 42. You 
can form a special evaluation team 
to do this task, use outside experts, 
or just pose the questions to your 
development team at critical 
checkpoints. Using the development 
team will keep this group of 
stakeholders involved throughout 
the adoption and implementation of 
the system. 



You should give special attention to 
monitoring the consequences of 
implementation. Since some 
systems have produced greater 
exclusion of students with 
disabilities from assessments, 
increases in special education 
referral rates, and increases in grade 
retention rates, it is important to 
monitor unintended consequences 
of implementation. 



Step 1 


Establish a Solid Foundation for Your Efforts 


Step 2 


Develop, Adopt, or Adapt a Model 


Step 3 


Establish a Data Collection and Reporting System 


Step 4 


Install the System 




Create incentives and support for adoption and use 




Prepare staff and the public for the changes 




■ Evaluate the system as it is implemented 
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Evaluation at Stage* of Implementation 

Prior to a final decision to implement system and allocate resources: 

Is our overall outcomes indicator system designed in a way that will be effective? Will we be able to produce the 
information that we want to produce, given our resources and the data collection, analysis, and reporting activities 
we have planned? Do the stakeholders agree with our plans? Do experts think our design meets standards? 



Basics in place, but have not proceeded with data collection and analysis: 

Are we ready to implement? Do we have the needed staff and are they prepared? Do we have the data processing 
equipment and materials we will need? Do we have the instrumentation? Is the climate right? 



Actively implementing: 

Are we being consistent with our plans? Are we using resources at the expected rate? Are we using staff time at 
about the estimated rate? Are we meeting our timelines? Are people satisfied so far? Are we storing and using data 
in proper ways? 



Through a complete cycle: 

Is the system having the desired effects on decision making? Did we accomplish our intended purpose? Are the 
results worth the effort? Are people satisfied? 



After each subsequent cycle: 

What have been the consequences of implementing an outcomes measurement? Do we need to reconsider any 
component of our system because of unintended consequences? Are there any other issues we need to revisit on 
any aspect of our system? 
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Some Final Thoughts 



As you follow the steps in this Self- 
Study Guide* you will encounter 
challenges that have not been 
mentioned here. The Resource 
Materials and Sources for Techn ical 
Assistance sections of this guide 
should be helpful. 



NCEO would be interested in 
hearing your comments, especially 
about your experiences in 
developing and implementing 
systems of outcomes and indicators. 
NCEO would also like to hear about 
unique uses of the NCEO outcomes 
model by states, districts, and local 
schools. 



Contact: 

NCEO 

350 Elliott Hall 

75 E. River Road 

Minneapolis, MN 55455 

612-626-1530 

Fax:612-624-0879 

Special Net: MN.OUTCOMES 

TDD: 612-624-4848 
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Each resource includes code 
symbols that correspond with the 
step(s) for which the resource will 
be most helpful: 

Step 1 - ♦ Establish a Solid 
Foundation for Your Efforts 

Step 2-% Develop, Adopt, or 
Adapt a Model 

Step 3 - A Establish a Data 
Collection and Reporting 
System 

Step 4 - ■ Install the System 



♦ • ▲ 

A Conceptual Framework for 
State Policy Development 

Samuelson, D., Elder, M. & Evans, 
J. (1990). Infants and Youne 
Children , 2 (3), 79-86. 



♦ • 

An Evolving Conceptual Model of 
Educational Outcomes for 
Children and Youth With 
Disabilities 

Ysseldyke, J., Thurlow, M., 
Bruininks, R., Gilman, C, Deno, S., 
McGrew, K., & Shriner, J. (August 
19, 1992). (Working paper 2). 
Minneapolis, MN: University of 
Minnesota, National Center on 
Educational Outcomes. 



♦ ■ 

An Overview of Planning Steps 
and Techniques 

Olsen, K. (1988). Lexington, KY: 
University of Kentucky, Mid-South 
Regional Resource Center. 

♦ • 

Consensus Building: A Process 
for Selecting Educational 
Outcomes and Indicators 

Vanderwood, M., Ysseldyke, J., & 
Thurlow, M. ( 1993). Minneapolis, 
MN: University of Minnesota, 
National Center on Educational 
Outcomes. 



♦ • 

Developing a Model of 
Educational Outcomes 

Ysseldyke, J., & Thurlow, M. 
(1993). Minneapolis, MN: 
University of Minnesota, National 
Center on Educational Outcomes. 



♦ • ▲ ■ 

Developing a System of 
Education Indicators: Selecting, 
Implementing, and Reporting 
Indicators 

Blank, R. K. (1993). Educational 
Evaluation and Policy Analysis . 15 
(1), 65-80. 



• ▲ 

Education Counts: An Indicator 
System to Monitor the Nation's 
Educational Health 

Special Study Panel on Education 
Indicators (Jul> 1991 ). Washington 
DC: National Center for Education 
Statistics. 



Educational Indicators 

Smith, M.( 1988). PhiJDelta 
Ka ppan , 69, 487-491. 



• ▲ 

Educational Indicators in the 
United States: The Need for 
Analysis 

Odden, A. (1990). Educational 
Researcher , 19 (5), 24-29. 



Educational Outcomes and 
Indicators for Early Childhood 

(Age 3) 

Ysseldyke, J., Thurlow, M., & 
Gilman, C. (August 1993). 
Minneapolis, MN: University of 
Minnesota, National Center on 
Educational Outcomes. 



Educational Outcomes and 
Indicators for Early childhood 

(Age 6) 

Ysseldyke, J., Thurlow, M., & 
Gilman, C. (August 1993). 
Minneapolis, MN: University of 
Minnesota, National Center on 
Educational Outcomes. 



Educational Outcomes and 
Indicators for Individuals at the 
Post-School Level 

Ysseldyke, J., Thurlow, M., & 
Gilman, C. (May 1993). 
Minneapolis, MN: University of 
Minnesota, National Center on 
Educational Outcomes. 
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RESOURCE MATERIALS (continued) 



Educational Outcomes and 
Indicators for Students 
Completing School 

Ysseldyke, J., Thurlow, M, & 
Gilman, C. (January 1993). 
Minneapolis, MN: University of 
Minnesota, National Center on 
Educational Outcomes. 



Effectiveness Indicators for 
Special Education: A Reference 
Tool 

National RRC Panel on Indicators 
of Effectiveness in Special 
Education (1986). Lexington, KY: 
University of Kentucky, Mid-South 
Regional Resource Center. 



Getting *he Word Out: Working 
With Your Local School Report 

Hennessey, A. (1992). Phi Delta 
Kappan, 73 (1), 82-84. 



• A 

Indicators for Monitoring 
Mathematics and Science 
Education 

Shavelson, R., McDonnell, L., & 
Oakes, J. (1989). Santa Monica, 
CA: Rand Corporation, Center for 
Policy Research in Education. 



♦ • ▲ ■ 

Issues and Options in Outcomes- 
based Accountability for Students 
With Disabilities 

Center for Policy Options in Special 
Education (1993). College Park, 
MD: University of Maryland. 



Issues in Constructing Program 
Theory 

Chen, H. in L. Bickman (Ed.) 
(1990). New Directions for Program 
Evaluation (Volume 47). San 
Francisco: Jossey-Bass. 



♦ • 

Organizing for Results: The Basis 
of Authentic Restructuring and 
Reform 

Spady, W. ( 1 988). Educational 
Leadership , 70, 504-511. 



♦ • ▲ 

Outcome Assessment in Special 
Education: Lessons Learned 

DeStefano, L., & Wagner, N. 
(1990). Transition Institute at 
Illinois, University of Illinois at 
Urbana-Champaign, and SRI 
International, Menlo Park, CA. 



♦ • 

Outcome Indicators for Special 
Education: A Model for Studying 
the Expected Outcomes of 
Education for Students With 
Disabilities 

Frey, W. ( 1991 ). East Lansing, MI: 
Center for Quality Special 
Education. 



Getting Things Done: How to 
Make a Team Work 

Hardaker, M., & Ward, B. (1987). 
Harvard Business Review , 
November-December, 112-117. 



A ■ 

How to Communicate About 
Restructuring 

Arnspbarger, A. ( 1991 ). State 
Education Leader . Summer issue, 
Denver, CO: Education 
Commission of the States. 



Managing Transitions: Making 
the Most of Change 

Bridges, W. (1991). Reading, MA: 
Addison- Wesley. 



• A 

Measuring Up: Questions and 
Answers About State Roles in 
Educational Accountability 

Office of Educational Research and 
Improvement (1988). Washington, 
DC: US Department of Education. 
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Program Evaluation: A 
Practitioner's Guide to Trainers 
and Educators 

Brinkerhoff, R., Brethower, D., 
Hluchyj, T., & Nowakowski, J. 
(1983). Design Manual . San 
Francisco, CA: Jossey-Bass. 



♦ ■ 

Restructuring the Education 
System: Building Private Sector 
and Community Support 

Education Commission of the States 
(1992). Denver, CO: Education 
Commission of the States. 
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RESOURCE MATERIALS (continued) 



♦ • ▲ ■ 

Special Education Program 
Evaluation: What Should States 
Consider? 

Olsen, K, & Massanari, C. (October 
1991). Lexington, KY: University 
of Kentucky, Mid-South Regional 
Resource Center. 



♦ ■ 

Stakeholder Participation and 
Utilization Program Evaluation 

Greene. J.G. (1988). Evaluation 
Review, 12 (2), 91-1 16. 



♦ ■ 

Stakeholders as Partners in 
Evaluation: A Stakeholder- 
Collaborative Approach 

Ayers, T.D. (1987). Evaluation and 
Program Planning . 10 . 263-27 1 . 



Using What Is Known About 
Change to Inform the Regular 
Education Initiative 

Loucks-Horsley, S., & Roody, D. 
( 1 990). Remedial and Special 
Education, JI (3), 51-56. 



♦ • ▲ 

Utilization-Focused Evaluation 

Patton, M. (1986). Beverly Hills, 
CA: Sage. 



♦ ▲ 

Views on Inclusion and Testing 
Accommodations for Students 
With Disabilities 

Ysseldyke, J., & Thurlow, M. (Eds.) 
(1993). Minneapolis, MN: 
University of Minnesota, National 
Center on Educational Outcomes. 



♦ ▲ ■ 

Standards for Evaluations of 
Educational Programs, Projects 
and Materials 

Joint Committee on Standards for 
Educational Evaluation (1981). New 
York. NY: McGraw-Hill. 



• ▲ 

What Work Requires of Schools: 
A SCANS Report for America 
2000 

SCANS (June 1991). Washington, 
DC: US Department of Labor, 
Secretary's Commission on 
Achieving Necessary Skills. 



♦ ▲ 

Testing Accommodations for 
Students With Disabilities: A 
Review of the Literature 

Thurlow, M. L., Ysseldyke, J. E., & 
Silverstein, B. (March 1993). 
Minneapolis, MN: University of 
Minnesota, National Center on 
Educational Outcomes. 
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SOTJRCES FOR TECHNICAL ASSISTANCE 



The U.S. Department of Education 
provides funds to support technical 
assistance to states and local 
education agencies. Many existing 
sources of technical assistance have 
access to expertise in the 
development of outcomes, 
assessments, and accountability 
systems. Some of these are 
identified here. 

REGIONAL EDUCATIONAL 
LABORATORIES 

Clients: State and local education 
agencies. 

Purpose: To help get the best 
available knowledge about 
improved practices and methods 
into the nation's classrooms, 
especially to foster and strengthen 
efforts to improve schooling for 
students at risk of leaving school 
unprepared for successful 
employment, further education or 
productive citizenship. 

Number of Centers: 10 

Funding: Office of Educational 
Research and Improvement (OERI) 

Contact for further information: 
OERI (202)219-2116 

Appalachia Educational 

Laboratory \ Inc. (A EL) 

1031 Quarrier Street 

P.O. Box 1348 

Charleston, WV 25325 

(304) 347-0400 

(800) 624-9120 (outside WV ) 

(800) 344-6646 (in WV) 

Terry L. Eidell, executive director 



Far West Laboratory for 
Educational Research and 
Development (FWL) 
730 Harrison Street 
San Francisco, CA 94107 
(415)565-3000 
Dean H. Nafziger, executive 
director 

Mid-continent Regional 
Educational Laboratory (McREL) 
Denver office: 

2550 S.Parker Rd., Suite 500 
Aurora, CO 80014 
(303) 337-0900 
Kansas City office: 
4709 Belleview Avenue 
Kansas City, MO 64112 
(816) 756-2401 

C.L. Hutchins, executive director 

North Central Regional 
Educational Laboratory (NCREL) 
1900 Spring Rd., Suite 300 
Oak Brook, IL 60521 
(708)571-4700 
Jeri Nowakowski, executive 
director 

Northwest Regional Educational 

Laboratory (NWREL) 

101 SW Main Street, Suite 500 

Portland, OR 97204 

(503) 275-9500 

Robert R. Rath, executive director 

Pacific Regional Educational 

Laboratory (PREL) 

1 164 Bishop Street, Suite 1409 

Honolulu, HI 96813 

(808) 532-1900 

John W. Kofel, executive director 



Regional Laboratory of 
Educational Improvement of the 
Northeast and Islands 
300 Brickstone Square, Suite 900 
Andover, MA 01810 
(508) 470-0098 
David P Crandall, executive 
director 

Research for Better Schools, Inc. 
(RBS) 

444 North Third Street 
Philadelphia, PA 19123 
(215)574-9300 

John E. Hopkins, executive director 

Southeastern Regional Vision for 
Education (SERVE) 
Headquarters: 

University of North Carolina at 

Greensboro 
P.O. Box 5367 
Greensboro, NC 27435 
(919) 334-321 1 or (800) 755-3277 
Field office: 

Office of Policy Research and 

Improvement 
Florida Department of Education 
325 West Gaines Street, Suite 414 
Tallahassee, FL 32399 
(904) 488-1611 

Roy H. Forbes, executive director 

Southwest Educational 
Development Laboratory (SEDL) 
21 1 East Seventh Street 
Austin, TX 78701 
(512)476-6861 

Preston C. Kronkosky, executive 
director 
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SOURCES FOR TECHNICAL ASSISTANCE (continued) 



EVALUATION ASSISTANCE 
CENTERS 

Clients: State Education Agencies 
(SEAs) and teachers and 
administrators in school districts 
with students in grades K-12 whose 
first language is not English. 

Purpose: Provide technical 
assistance in program evaluation 
and student assessment under the 
Bilingual Education Act (ESEA, 
Title VII). 

Number of Centers: 2 

Funding: Office of Bilingual 
Education and Minority Language 
Affairs (OBEMLA) 

Contact for further information: 

OBEMLA-(202) 205-5463 

Evaluation Assistance Center - 
Eastern Region (EAC-EAST) 
The George Washington 
University 

1730 North Lynn Street, #401 

Arlington, VA 22209 

(703) 528-3588 

Fax: (703) 528-5973 

Charlene Rivera, director 

Intercultural Development 

Research Association 

5835 Callaghan, #350 

San Antonio, TX 78228 

(512)*684-8180 

Fax: (512) 684-5389 

Albert Cortez, associate director 



Evaluation Assistance Center 
Western Region (EAC-WEST) 

New Mexico Highlands University 
121 Tijeras NE, Suite 2100 
Albuquerque, NM 87102 
(800) 247-4269 
(505)242-7447 or 7472 
Fax:(505)242-7558 
Paul Martinez, director 



REGIONAL RESOURCE 
CENTERS 

Clients: State Education Agencies 
(SEAs), and through those SEAs, to 
other state and local agencies in 
providing early intervention, special 
education, and related services to 
infants, toddlers, children, and 
youth with disabilities and their 
families. 

Purpose: Increase agency capacity 
to meet the needs of infants, 
toddlers, children, and youth with 
disabilities and their families. 

Number of Centers: 6 regional 
centers plus a federal center for 
coordination 

Funding: Office of Special 
Education Programs (OSEP) 

Contact for further information: 

OSEP (202) 205-8007 
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Great Lakes Area Regional 
Resource Center (GLARRC) 
The University of Ohio 
700 Ackerman Road, Suite 440 
Columbus, OH 43202 
(614) 447-0844 
Fax:(614)447-9043 
TDD: (614) 447-9186 
Larry Magliocca, director 

Mid-South Regional Resource 
Center (MSRRC) 

University of Kentucky 

1 14 Mineral Industries Building 

Lexington, KY 40506 

(606) 257-4921 

Fax: (606) 258-1901 

Robert L. Sterrett, director 

Mountain Plains Regional 
Resource Center (MPRRC) 

Utah State University 

1780 North Research Parkway, #112 

Logan, UT 84321 

(801) 752-0238 
Fax:(801) 753-9750 
Glenn Latham, director 

Northeast Regional Resource 
Center (NERRC) 
Trinity College of Vermont 
Burlington, VT 05401 

(802) 658-5036 
Fax: (802) 658-7435 
TDD: (802)860-1428 
Special Net NERRC 
Edith Beatty, director 

South Atlantic Regional Resource 
Center (SARRC) 

Florida Atlantic University 
1236 North University Drive 
Plantation, FL 33322 
(305)473-6106 
Fax: (305)424-4309 
Timothy C. Kelly, director 
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SOURCES FOR TECHNICAL ASSISTANCE (continued) 



Western Regional Resource Center 
(WRRC) 

University of Oregon 

Center on Human Development 

Eugene, OR 97403 

(503) 346-5641 

Fax: (503) 346-5639 

TDD: (503) 346-5641 

Richard W. Zeller, director 

Federal Resource Center for 
Special Education 
Academy for Educational 
Development 

1255 23rd Street NW, Suite 400 

Washington, DC 20017 

(202)862-1487 

Fax: (202) 466-8693 

Carol H. Valdivieso, director 



Center for the Study of Testing, 
Evaluation, and Educational 
Policy (CSTEEP) 
(Funded by the National Science 
Foundation.) 
323 Campion Hall 
Boston College 
Chestnut Hill, MA 02167 
(617) 552-4521 
Fax: (617)552-8419 
George F Madaus, principal 
investigator 



SPECIAL ASSESSMENT 
PROJECTS 

National Center for Research on 
Evaluation, Standards, and 
Student Testing (CRESST) 
(Funded by the Office of 
Educational Research and 
Improvement, U.S. Department of 
Education.) 

UCLA Graduate School of 

Education 
405 Hilgard Avenue 
Los Angeles, CA 90024 
(310) 206-1532 
Fax: (310) 825-3883 
Eva L. Baker and Robert L. Linn. 

co-directors 
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NCEO works in collaboration with 
St. Cloud State University and 
National Association of State Directors of Special Education 



ERLC 



